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Data-driven decisionmaking has never been more important in education — particularly 
in Career Technical Education (CTE). COVID-19 (the coronavirus) has had widespread and 
unprecedented impacts on the educational system and economy. 


To support access to high-quality education and prepare 
learners for economic success during COVID-19 and beyond, 
CTE leaders and other policymakers need access to valid, 
reliable data to improve the quality and availability of program 
offerings and ensure that the needs of each learner are met. In 
addition, learners and their families need accurate information 
about educational and employment opportunities to make 
decisions about education and training options, including 

the need for upskilling and reskilling driven by the pandemic. 
Likewise, employers need information on the local and 
regional supply of skilled workers and their credentials and 
competencies. All of these needs point to the importance of 
full and accurate reporting on CTE programs and the value of 
data that is meaningful — and accessible — to a wide variety 
of audiences. 


At the same time, CTE leaders are working to implement the 
new Strengthening Career and Technical Education for the 
21st Century Act (Perkins V), which includes a strong focus on 
sharing data with learners, families and other stakeholders as 
well as using data to improve programs and make decisions 
about funding priorities. To help address the challenge of 
reporting meaningful data, Advance CTE, with support from 
the Association for Career and Technical Education, convened 
a Shared Solutions Workgroup of national and state leaders to 
identify promising practices and develop recommendations 
to improve the sharing and use of CTE data with internal and 
external stakeholders under Perkins V. This paper can serve 
as a roadmap for state and local leaders as they develop and 
redesign public reporting mechanisms — including report 


cards, fact sheets, infographics, Perkins accountability reports, 
dashboards and more — that not only meet the requirements 
of the law but also align with broader efforts for data-driven 
decisionmaking and meet the needs of end users. 


Perkins V Reporting Requirements 


While previous iterations of the Perkins legislation required 
local public reporting of CTE data, Perkins V builds on this 
foundation by requiring states to publicly report data as 

well. In addition, the law puts greater emphasis on evidence- 
based decisionmaking and provides more detailed reporting 
instructions, elevating the importance of making data 
available and accessible to help all stakeholders interpret and 
use it to inform action. 


As in the past, under Perkins V, each state must report to the 
U.S. Department of Education the indicators of performance 
laid out in the law. In this federal reporting, states must 
disaggregate data by race/ethnicity, gender, migrant 

status (secondary only), and the special population groups 
outlined in Perkins V (a list that is expanded from Perkins IV). 
States now must also disaggregate federal data by the CTE 
program/program of study or Career Cluster® to drill down 
to more specific program-level performance. These federal 
reports must intentionally identify and quantify any gaps in 
performance between groups of learners. The secretary of 
education will then make this information available to the 
general public as well as to Congress in reports, including 
state-by-state comparisons. 
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@ Individuals with disabilities; 


PERKINS V DEFINES 
THE FOLLOWING 
SPECIAL 
POPULATIONS:' 


low-income youth and adults; 


® Individuals preparing for non-traditional fields; 


® Single parents, including single pregnant women; 


© Out-of-workforce individuals; 


In addition to the required federal reporting and national 
dissemination of CTE data by the U.S. Department of 
Education, Perkins V goes further than previous iterations 
of the law to require states to disseminate this information 
more broadly. States must publicly report the performance 
of all CTE students and subgroups of learners and make 
this information available “widely, including to students, 
parents, and educators; through a variety of formats, 
including electronically through the Internet; and in user- 
friendly formats and languages that are easily accessible, as 
determined by the eligible agency.” 


Similarly, Perkins V requires local grant recipients to report 
CTE performance data to the state and disseminate data to 
learners, families, practitioners and the public in a format 
that is “understandable and uniform” and “as practicable, 
provided in a language that students, parents and educators 


PRECONDITIONS 
FOR SUCCESS 


Cluster. 


States and local recipients must have a foundation of accurate, timely data collection from which to build data tools that will inform CTE 
stakeholders and support program quality. These preconditions for success include: 


@ Valid, reliable and complete data that is collected consistently; 


Meeting this last precondition will be easiest if the state can collect learner-level data instead of aggregated data. Another important step 
toward meeting these preconditions for success and ensuring consistency across CTE data collection is developing data dictionaries and business 
rules that set parameters for collecting and reporting data. 


In addition, state leaders should strive to align terminology, definitions and common measurement approaches across federal programs and 
with other state agencies. While the Perkins eligible agency alone cannot accomplish this alignment, cross-agency conversations can begin to 
move state data in a more consistent direction and go a long way toward creating more useful, easy-to-understand public reporting. 


English learners; 


@® Individuals from economically disadvantaged families, including © ® Individuals experiencing homelessness (as described in the 


McKinney-Vento Homeless Assistance Act); 
@ Youth who are in or have aged out of the foster care system; and 


@® Youth with a parent who is a member of the armed forces and on 
active duty (as defined in the United States Code). 


can understand.’ While these local requirements are similar 
to the expectations in Perkins IV, the language is more specific 
and focused on ensuring that all stakeholders have access to 
meaningful data. 


Perkins V sets the minimum expectations for reporting CTE 
data, but states can and should go beyond the letter of the 
law and use this moment in time as a springboard to enhance 
the quality and availability of CTE data. Already, some states 
are identifying additional metrics and performance indicators 
that are not required by federal law but will be useful to 
practitioners and the public. Others are designing new, user- 
friendly dashboards and reports that equip key stakeholders 
to understand the data and take action. As states align their 
existing reporting to meet new expectations in Perkins V, they 
should seize the moment to produce tools and resources that 
will be useful to their key constituents for years to come. 


@ Data from the most recent timeframe appropriate for a particular indicator; and 


@ Data that can be disaggregated by all required learner groups and special populations, as well as by CTE program/program of study or Career 
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Design Principles for CTE Data Reporting 


While states and local Perkins grant recipients have discretion in how to present their CTE data, Perkins V gives them the 
opportunity to make data more accessible, more user friendly and easier to interpret through a concerted effort to improve 
data reporting and dissemination tools. The following design principles can guide state leaders in disseminating data that is 
purposeful, visually compelling and understandable. Each principle includes a list of key questions to consider to help state 
leaders develop an action plan for creating impactful data tools and visualizations. 


1. Clarify the Purposes for Sharing Data The principles are intended to help CTE leaders design public 

2. Make Data Easy to Find data reporting and dissemination tools that not only meet 

3. Make Data Visually Appealing the specific requirements in Perkins V but also go beyond the 

4. Clearly and Consistently Label and Describe Data minimum requirements. Going beyond the minimum can 

5. Make Data Accessible ensure that all relevant stakeholders have the information 

6. Disaggregate Data to Highlight Equity they need when they need it and that they can access and 

7. Provide Context to Add Meaning interpret it with ease. These principles can apply to a variety 

8. Enable Interactivity and Customization for Key of informational and promotional tools; equity-focused 
Audiences resources; and local, state or federal required reporting. 


9. Help Users Interpret Data and Take Action 


1. Clarify the Purposes for Sharing Data 


For example, if a purpose for sharing data is to help learners 
and, when appropriate, their families make more informed 
choices about enrolling in CTE programs, then including 
program-level data and highlighting labor market outcomes 
in public reports is important. If the primary purpose is an 
aggregate evaluation of programs for the purposes of state 
policy or funding, other elements may be more important. 
Some potential purposes for data sharing include: 


KEY 
QUESTIONS 
TO CONSIDER 


What are your objectives for CTE data reporting and 
dissemination? 


What other reporting and communications tools can 
supplement CTE data reporting? 


How have you engaged relevant stakeholders in 
identifying the purposes of CTE data reporting and 
the appropriate reporting and communications tools? 


© Providing accountability/public transparency; 


. : . ; © Informing practitioners; 
To provide meaningful data that is accessible and useful, state Ze 


leaders should clarify their purposes for sharing data and © Supporting program improvement, 


for each decision that is made in the presentation of data. © Informing policy and budget decisions; 
Developing this clear purpose will require internal clarification © Providing career guidance; 
about the state CTE agency’s objectives, discussion with © Highlighting equity issues; 


multiple stakeholders about their needs and goals for using 
data, and research into best practices for ensuring that the 
data is accessible to the intended audiences. Even the process 
of defining audiences and clarifying the purposes of CTE data 
can lead to more focused and actionable reporting. 


© Evaluating program effectiveness; 
© Informing employers; and 
© Spurring action among CTE advocates. 


FFB TE 
Beyond the Numbers: Design Principles for CTE Data Reporting 3 


In many cases, public data reporting will need to serve 
multiple purposes, and documenting these purposes at 

the beginning of the design process will help to guide 
decisionmaking along the way. At the same time, states should 
consider the appropriate reporting tools for communicating 
data. For purposes of this report, data tools could include but 
are not limited to: 


© High school or college report cards; 


© Issue-specific reports, such as publications addressing 
equity or career readiness; 


© Interactive dashboards; 

© Perkins accountability reports; 

© Fact sheets, infographics and brochures; 
© Presentation slides; 

© Spreadsheets; 

© Videos; 

© Data stories; or 


© Social media and e-newsletters. 


2. Make Data Easy to Find 


One single reporting and dissemination tool will likely not be 
able to meet all the data-related goals in a particular state. 
State leaders should consider using multiple reporting tools 
— in various formats — to communicate relevant information 
to different audiences and help meet the needs of every CTE 
stakeholder for meaningful data. 


The Shared Solutions Workgroup’s accompanying 
communications toolkit, Beyond the Numbers: A Toolkit for 
Communicating CTE Data, can help states identify target 
audiences and the information in which those audiences 

are most interested. The toolkit also provides guidance, best 
practices and customizable tools to help state leaders design 
communications that integrate data visualizations, text and 
other elements to tell a story and encourage action from 
targeted audiences. 


KEY 
QUESTIONS 
TO CONSIDER 


How easily can users find CTE data tools by 
navigating state CTE websites or through a Google 
search? Have you engaged stakeholders in user 
testing? 


How clearly are CTE data tools labeled in website 
menus? 


Are secondary and postsecondary CTE data available 
in the same place? 


How well is CTE data integrated with other state 
data? If CTE is integrated into Every Student Succeeds 
Act (ESSA) report cards or postsecondary datasets, 
how easy is it for users to navigate from the state CTE 
website to these other reporting tools? 


To ensure wide access to data, state leaders must make data 
tools easy to find on state CTE websites. Too often, current 
reporting is buried on website subpages, not clearly labeled in 
menus, or found in several different places. Data tools should 
be centralized on state CTE agency websites under a clear 
label such as “Data and Results” or “CTE Accountability Reports” 


to help all stakeholders find them. Additionally, secondary and 
postsecondary CTE data should be available in the same place 
so that linkages can be made between the two. 


In addition to creating an easy-to-find, centralized home for 
both secondary and postsecondary CTE data on state CTE 
websites, states should include CTE performance data within 
other state education data reporting and tools such as state 
ESSA report cards or state community and technical college 
reporting. If CTE data is incorporated into these other state 
data sources or available on other agencies’ sites, links to these 
other data reports should be placed prominently on the CTE 
website. 


States can harness website search engine optimization to 
ensure that materials are easy to find, followed by user testing 
to check whether the appropriate website comes up on the 
first or second page of a search engine query.’ Using clear 
language and key terms to describe reporting tools and 
locating them prominently on the state website can help 
boost their placement in search engine results. State leaders 
can also ask stakeholders to test website functionality to 
ensure that they can quickly and accurately locate data. 
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3. Make Data Visually Appealing 


KEY 
QUESTIONS 
TO CONSIDER 


What is an appropriate visualization based on your 
purpose and data type? 


How can you use lines, shapes, color, text and layout 
to draw readers in and hold their attention? 


How can you use lines, shapes, color, text and layout 


to direct readers’ attention to key information? 


How can you employ a style sheet to create 
consistent, user-friendly visualizations? 


Have you engaged stakeholders in developing easy- 
to-read CTE visualizations? 


To help users make sense of the numbers, CTE data should 
be presented in ways that are visually appealing and easy 

to follow. Many types of visualizations can make data more 
engaging and easier to read, including charts, heatmaps and 
pictograms. Examples of these visualizations are provided on 
pp. 6-7. 


Regardless of the type of data visualization chosen, it should 
make strategic use of the basic elements of visual design, 
such as lines, shapes, color, text and layout.*° A successful 
visual design balances unifying design elements, such as the 
repetition of a particular font or color, with contrast and visual 


variety. Design experts Evergreen and Emery also recommend: 


© Data that is ordered for the viewer (for instance, by 
frequency or time period); 


© Horizontal labels positioned near the data; 
© Hierarchical text size (larger titles and smaller labels); 
© Text color that contrasts with the background color; and 


© Sparing use of borders, gridlines and axis lines.’ 


Employing basic design principles can signal to users how to 


general public. For example, coloring a data element green to 
indicate that a local recipient is meeting goals or has improved 
from the prior year can quickly show users where performance 
is strong. Similarly, directional arrows and checkmarks can 
quickly signal to readers where performance is increasing or 
decreasing or where objectives have and have not been met 
(these symbols also have the added value of being accessible 
for users with color blindness). State leaders can share drafts 
of visualizations with a selected subset of stakeholders to 
ensure that they are easy to read and interpret and that they 
are accessible for a variety of users (more information on 
accessibility can be found in Principle 5). 


Statewide Proficiency Rates 


English Language Arts 


2016-17 2017-18 2018-19 
62.1% 63.7% 64.6% A 
Mathematics 
2016-17 2017-18 2018-19 
60.2% 60.4% 61.0% A 
Science 
2016-17 2017-18 2018-19 
66.6% 70.0% 69.1% V 


Source: Ohio Department of Education’ 


In addition, state CTE leaders can develop a style sheet that 
defines how all visualizations should look by specifying 
colors, fonts, label sizes and other visual elements so that data 
reporting and dissemination tools are consistent and well 
designed.’ This style sheet may apply just to CTE reporting or 
may be aligned across state education systems. 


read a data visualization and make data more accessible to the 


TF i 
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There are a nearly infinite number of data visualization types and many different ways to name and categorize these techniques. 
CATEGORIES . era F 
air The examples below introduce several visualizations that state leaders can harness to represent CTE data and list the best uses 
VISUALIZATION for each type of visualization. These visualizations are possible with different types of software or levels of technical capacity 
and can be incorporated within a variety of reporting tools, including infographics, spreadsheets, webpages, dashboards, 
reports and presentation slides. The additional resources listed on p. 15 provide further information on data visualization types 
and the circumstances in which they are best used. 


BAR CHARTS 
e Compare groups or categories of data 


e Use grouped or stacked bars for two or more categories 


POVERTY Percentage of high school students who were CTE participants 
77 percent of high school students were CTE participants 
The poverty rate among Z (earned at least one credit in CTE) 

Rhode Island's young adults White i 
is slightly higher than that Two or more races 
of the nation, and falls in Black 
the higher range of New 
England states. 


Hispanic 

Asian 

American Indian/Alaska Native 
Pacific Islander 


Rhode Island Maine 
Male 


(RN Ator Above Poverty Level NNN Below Poverty Level Female 


Source: Ri DataHUB” Source: U.S. Department of Education" 


LI NE CHARTS Five Year Trend for Kentucky High School Graduates' College 
and/or Career Readiness  % College and/or Career Ready 


e Show change over time 100% Il % Career Ready 
™ % College Ready 
e Combine with a bar chart to make a dual axis or combination chart 90% 


Source: KYSTATS!2 80% 


70% 
60% 


50% 


n 
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40% 


30% 


20% ous 
% 


10% 
0% 
2014 


Academic Year 


PIE CHARTS 


e Depict groups or categories of data as part of a whole 
(i.e., all categories add up to 100 percent) Bm African American 5% 


e Work best with relatively simple datasets © American Indian 89% 


Source: Wisconsin Department of Public Instruction” @ Asian 3.46% 


@ Hispanic 10% 
(| Pacific Islander 0.06% 
™@ Two or More Races 2.74% 


i White 78% 


Beyond the Numbers: Design Principles for CTE Data Reporting 6 


CATEGORIES 
OF DATA 
VISUALIZATION 


PICTOGRAMS/UNIT CHARTS Oe ee eT ee 


* Use icons to represent quantities 98% of students graduate from high school 


¢ Work best with relatively simple datasets pararapapars 
Source: Ohio Department of Education" 

60% of students obtain further education 
ee eoeeeecses @ @ 


50%-+ of students are employed within 9 months of graduation 


TREEMAPS Production Office & Sales & Construction 
Administrative Related & Extraction 
Support Occupations Occupations Occupations 


e Nest proportionally sized shapes within a larger shape i aszh an 


Maintenance, & 


e Show the relationship between parts of a whole TTT LATS 


Source: Data USA" 
Occupations 


Apprenticeship Enrollments for Graduates in 2018 


MAPS 
e Show geographical data 
Source: KYSTATS"® 


HEATMAPS Percent Change in Employment 


e Use color to define categories 


e Can be applied to other visualization types, such as tables 
and maps 


Source: Opportunity Insights” 


Ea im 
“70% -56 -42 -28 -14 0 +14 42.8 +4.2 +5.6 +7% 
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4. Clearly and Consistently Label and Describe Data 


KEY 
QUESTIONS 
TO CONSIDER 


Do CTE data tools and visualizations incorporate clear 
labels? 


Are CTE indicators and population groups, as well as 
data sources and limitations, defined in a way that 
makes sense to non-expert stakeholders? Have you 
engaged stakeholders to ensure that definitions are 


understandable to a lay audience? 


How easily can expert users find additional technical 
information on indicators or data sources? 


Are the same definitions used consistently across CTE 
reporting? How well aligned are CTE data definitions 
with other state reporting? 


One of the most important ways to help stakeholders 
understand CTE data is to provide a clearly organized and 
labeled resource with easy-to-understand descriptions. Each 
visualization should have a brief title. Web design research 
has found that readers, particularly English readers, start at 
the upper left, so that is a prime location for placing a short, 
descriptive title.'® Data elements within visualizations, such 
as rows and columns, should also have clear labels. If a report, 
dashboard or similar resource includes multiple visualizations, 
they should be organized so that readers are first introduced 
to high-level data before digging into specifics." 


As part of clearly labeling data, state leaders should strive to 
create user-friendly names and descriptions, avoiding jargon 
and “eduspeak” and minimizing acronyms. Avoiding this 
language can be particularly challenging when it comes to 
defining Perkins V indicators and special population groups. 
Oftentimes CTE leaders use the esoteric numbering system 
developed at the federal level to refer to specific Perkins 
indicators, such as “5S2” to refer to the secondary CTE program 
quality indicator that measures attainment of postsecondary 
credits or “451” to refer to non-traditional program enrollment. 
This terminology is likely to confuse a non-expert audience. 


Instead, state CTE leaders should work with their 
communications teams or communications personnel in their 
agency to develop names and descriptions for indicators, 
population groups and other relevant CTE-specific data using 
commonly understood terms. For instance, the Wyoming 
Department of Education includes brief, descriptive examples 
in its 4S1 indicator definition to help users understand 

what is meant by non-traditional: “The percentage of CTE 


concentrators in career and technical education programs 
and programs of study that lead to non-traditional fields, i.e. 
female students in auto technology, male students in health 
occupations, etc.’”° 


Labels and descriptions can be tested with a variety of 
stakeholder audiences to ensure understanding. These 
user-friendly labels and definitions should be incorporated 
directly into data tools and visualizations, not buried on 
separate webpages. Links, popout boxes or “tooltips” — text 
that appears when a user's mouse or touchpad hovers over a 
particular webpage element — that provide more technical 
definitions can provide additional details for an expert 
audience. (To ensure accessibility, links, popout boxes and 
tooltips should be screen-readable and viewable on mobile 
devices. See Principle 5 for more on accessibility.) 


Data tools and visualizations should also clearly identify data 
sources to enhance understanding and provide transparency. 
This includes identifying the names of state data systems, time 
ranges for the data presented, and the presence of missing 
data or data that is suppressed to protect student privacy when 
a population group, program and/or institution includes a small 
number of learners, as described further in Principle 6. Shorter 
descriptions of data sources with less technical terminology 
can be integrated directly into the visualization, with links to 
more technical documentation for those users who need it. 


Once clear and concise descriptions of indicators, population 
groups and data sources are developed, states should use 
them consistently across state and local CTE reporting. Using 
these descriptions and terminology consistently across 

other state data tools is helpful as well. Ideally, common 

data standards, data dictionaries and terminology for 
credentials and competencies across the state can support 
this consistency. If consistency cannot be accomplished, states 
should craft data reporting and dissemination tools with labels 
and descriptions that point out the differences. For example, if 
the graduation and extended-year graduation rates reported 
under Perkins V do not match the graduation rates presented 
in other state materials, owing to differences in federal and 
statutory reporting requirements, clear terminology to 
differentiate between the two is important. If one report 
includes a standard cohort graduation rate while another 
includes this cohort as well as learners who earn a graduate 
equivalency degree or certificate of completion, a more exact 
descriptor than “graduation rates” should be used in each 
report to address the nuances reflected. 
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5. Make Data Accessible 


KEY 
QUESTIONS 
TO CONSIDER 


Which populations do you need to engage to test if 
CTE reporting and visualizations are accessible? 


Can your data tools and visualizations be read by 
screen readers through alternate text or similar tools? 


Is data reporting available in languages other than 
English? Were translations produced or verified by 
bilingual experts? 


Is CTE data available in multiple web- and print- 
based formats? 


How easily can data tools and visualizations be 
accessed and read on mobile devices? 


To ensure that all users can read and understand CTE data, 
states should design “at the margins” by beginning with the 
needs of users who will have the most barriers to access. 
Stakeholder engagement at the outset, and user testing in 
later stages, can help to ensure accessibility. The end result of 
this design process will be accessible not only to users who are 
traditionally under-served but also to the general public. 


Prioritizing accessibility includes making data tools that are 
easy to read and understand; available in multiple formats 
and languages; and usable by assistive devices, particularly 
screen readers. In addition to being best practice, accessibility 
may also be legally required: The Americans with Disabilities 
Act prohibits state and local government agencies from 
discriminating in access to services, programs and activities, 
which can be interpreted to include data reporting, and many 
states have passed specific laws requiring accessibility in state 
and local agency publications and websites.”! 


Design elements that reduce visual clutter, eliminate jargon 
and direct readers’ attention, as described further in Principles 


3 and 4, can help all readers, particularly individuals with 

low vision and individuals for whom English is not their first 
language. Best practices include legible sans serif fonts, color 
contrast, and the use of dashed and dotted formatting in 

line graphs to help readers differentiate one trend line from 
another.?3 In addition, software features such as alternate 
text that translate visualizations into words for screen readers 
can further support accessibility for stakeholders who are 
blind or have low vision. 


State agencies should also translate data reports into 
languages other than English to ensure that non-English- 
speaking stakeholders can access information. This translation 
should involve more than plugging text into an online 
translator; rather, a bilingual speaker should generate or verify 
any translation to ensure its accuracy. 


Beyond meeting the needs of people with disabilities and 
non-English speakers, providing the data in multiple formats 
can also expand access by enabling stakeholders to use this 
data in different contexts. These formats include reporting and 
dissemination tools that can be downloaded for additional 
analysis, printable PDFs that can be shared at in-person 
meetings, and online data sources designed with mobile 
devices in mind. 


In addition to stakeholder engagement and usability 

testing, accessibility guidelines from trusted sources such as 
Section508.gov and software tools for testing accessibility 
can facilitate the design process. One quick and easy tool is 
the NoCoffee extension for Google Chrome, which simulates 
a variety of visual impairments. The additional resources 
listed on p. 15 provide further information on accessibility for 
different types of disabilities and accessibility testing tools. 
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6. Disaggregate Data to Highlight Equity 


KEY 
QUESTIONS 
TO CONSIDER 


In public reporting, are you suppressing data when 
group sizes are smaller than allowed by your state 
and/or recommended by experts? 


Can CTE data be averaged over multiple years to 
protect student privacy and show trends? 


Can CTE data be disaggregated by more than one 
population category at a time? 


Can practitioners securely access disaggregated data? 


Do reporting and dissemination tools use asset-based 
language? Have you communicated with special 

and under-served groups about their preferences for 
language used in reporting? 


Disaggregating, or separating, data to show trends for 
different learner groups can shine a light on opportunity gaps 
and help educators, policymakers and other stakeholders 
collaborate to close gaps and improve equity. It is imperative 
that state CTE leaders disaggregate data to better identify 
equity issues within CTE and share this data to build trust 
through transparency. Furthermore, Perkins V requires states 
and local grant recipients to disaggregate data by gender, 
race/ethnicity, migrant status (secondary only), and the special 
population groups identified on p. 2, as well as by program 

of study or Career Cluster, and to share data with the public 
on the CTE participation and performance of each of these 
learner groups. 


When disaggregating data, state CTE leaders should take 

care to present this information in ways that protect student 
privacy. Many experts recommend publicly reporting state- 
level data only if the group includes at least 10 students, with 
data on smaller groups hidden.” Data scientists refer to group 
size in this context as “n-size,”and maintaining minimum 
n-sizes can ensure that performance data that could be linked 
to individual learners is not made public. It also prevents users 
from trying to draw conclusions from data based on very small 
sample sizes. 


State leaders should consult state law or regulation, as well as 
district or institutional practices, to determine the appropriate 
n-sizes. One way to protect student privacy and help users 
derive more meaning from disaggregated data is to use 

data averaged over three years to create larger cohorts. This 
practice not only protects student data but also filters out 
year-to-year “noise” or minor discrepancies in data to better 


illustrate longer term trends, without eliminating data on 
smaller population groups. 


While disaggregation by single population categories can 
pinpoint important equity trends, state leaders can go deeper 
with multi-layered disaggregation. No one is represented by 
just one identity, so disaggregating by population group as 
well as program area, or by multiple population categories, 
can be meaningful. Educators, policymakers and other 
stakeholders can develop more specific, real-world strategies 
for closing gaps when considering data on Black female 
learners rather than data on Black learners and female learners 
as separate groups or when considering the participation of 
learners with disabilities in particular program areas rather 
than in CTE overall. However, this technique can also result 

in very small group sizes, as noted previously, and every 
combination may not be appropriate for public reporting. 


Whichever approaches the state takes to disaggregation in 
public data tools, CTE practitioners should have access to 
learner-level data for the students they work with, without 
suppression of small numbers, to enable them to identify 
opportunity gaps and devise solutions. These data tools and 
visualizations should be housed on secure portals and require 
appropriate credentials to access. Disaggregated data, even 
when unavailable to the general public, can still be visualized 
using the design principles incorporated in this document 

to help practitioners — most of whom are not data scientists 
— to understand the data and use it as the basis for program 
evaluation and improvement. 


Whether they are reporting data publicly or creating secure 
data tools for practitioners, state CTE leaders should avoid 
presenting data on the performance of different population 
groups and comparisons between groups in a way that 
“others” certain groups of students or centers learners of 
privilege as the baseline CTE participants. Using asset-based 
rather than deficit-based language in disaggregated or equity- 
related data can help address this issue. For example, using 
“opportunity gap” instead of “achievement gap” can illustrate 
that disparities result from systems-level factors rather than 
learners’ individual abilities.?> 7° People-first language is also 
preferred by many stakeholders from special population 
groups — for instance, using “learners with autism” rather than 
“autistic learners” — although this preference is not universal. 
Communicating with stakeholders from particular learner 
groups about their data needs and language preferences can 
improve data tools. 
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7. Provide Context to Add Meaning 


KEY 
QUESTIONS 
TO CONSIDER 


How many years’ worth of CTE data can you use to 
show trends over time? 


What non-CTE data, such as labor market information 
or data from all learners in the state, is available that 
can help provide context for users? 


How well does non-CTE data align to CTE data? How 
comparable are the student groups, time periods 
and definitions used? How can you communicate 
differences between CTE data and additional data 
that is used to provide context? 


One or two years of CTE data, in isolation, may be insufficient 


to help stakeholders understand the impact of CTE on learners. 


State leaders can enrich data reporting by using multiple years 
of CTE data as well as other related datasets to add context 
and help make CTE data more meaningful. Several types of 
data can better demonstrate the impact of CTE by showing 
trends over time and provide useful comparisons: 


© Historical/longitudinal data; 

© Performance of all learners in the state; 

© Progress toward meeting statewide career readiness 
goals, such as a postsecondary credential attainment 
goal; and 

© Labor market data. 


For example, reporting the credential attainment rate of 
postsecondary CTE students on its own may not mean much 
to users, especially learners and families. Adding comparisons 
to the credential attainment rate for all learners or a state's 


credential attainment goals, if available, can help to provide 
important context to CTE data. 


Any additional data points used to provide context must be 
truly comparable to CTE data, or distinctions between CTE 
data and other data must be clearly noted. If definitions, 
populations, time periods or measurement approaches used 
in non-CTE data are too different, offering comparisons may 
skew interpretations and reduce the validity of reporting. For 
example, some labor market information applies to regions 
of a state that do not align with school districts or community 
college service areas, so caution should be taken when 
building comparisons. 


The more comparable any additional datasets are, the more 
they can be integrated directly into visualizations with CTE 
data. For instance, a chart showing the graduation rates of 
CTE concentrators could include a column illustrating the 
graduation rate of all learners in the state, if the cohorts and 
time periods match. Data that is relatively, but not exactly, 
Postsecondary Remediation Trend: 
CTE Students vs. All Students 


354 31% 
28% 28% 


30% 
2596 23% 
20% 18% 
aoe 11% 
10% kis 7% 
i 
0% 


2011-2012 2012-2013 2013-2014 2014-2015 


@CTEConcentrators m™All Students 


comparable can be communicated through a separate 
visualization or additional text. 


Source: Fleck Education in partnership with Thomas P. Miller & Associates” 
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8. Enable Interactivity and Customization for Key Audiences 


KEY 
QUESTIONS 
TO CONSIDER 


What questions might different users ask about 
CTE data? Have you consulted with stakeholders 
about their most pressing questions and needs for 
customization? 


What tools do you have that can enable users 

to customize data, such as dashboards or Excel 
spreadsheets with filtering? What technical, financial 
and data security considerations affect your use of 
these tools? 


Where can users go to find more information and other 
data tools that answer their questions? 


State CTE leaders can go beyond providing context to 
enabling users to customize or filter data for their particular 
needs, locations or interests. Incorporating interactive 
elements ensures that multiple audiences can find individual 
meaning in the data. Customization should be limited to 
filters that answer stakeholders’ most frequent and important 
questions, as too many customization options can overwhelm 
the user. 


Filtering by Career Cluster or program of study, as well as 

by local education agency or postsecondary institution, 

is a common customization option for CTE data. Allowing 
users to customize data locally serves a dual purpose of 
answering questions about CTE in their district or college and 
meeting local reporting requirements. It also helps to ensure 
consistency and validity in local reporting across the state. 
Additional filtering options may also be helpful, depending 
on the data being presented; state CTE leaders can engage 
stakeholders to determine which options for customization 
would be most impactful for different audiences. 


Dynamic and interactive data dashboards — which states 
can develop using pivot tables and spreadsheets or through 
proprietary platforms such as Tableau, Microsoft Power 

Bl and similar software — are a best practice in creating 

data reporting that is informative for multiple audiences. 
These tools frequently integrate filters, dropdown menus or 
checkboxes that enable users to customize reports by Perkins 
indicator, time period, population group, Career Cluster/ 
program of study, or local school district/postsecondary 
institution. State CTE agencies do not need specific software to 
implement interactivity: Excel spreadsheets can be filtered to 
enable customization. 


Whether the state chooses a platform like Tableau or Microsoft 
Power BI or uses Excel or a similar spreadsheet tool will depend 
on the desired audiences as well as technical and financial 
considerations. Cloud-based dashboards require an ongoing 
financial investment and restrict user access to source data, 
securing student information and facilitating public reporting. 
Filtered spreadsheets require less of an investment financially 
but frequently display source data within the spreadsheet 
itself, making these tools more suited for non-public purposes, 
such as when practitioners are analyzing data for program 
improvement. 


Whatever customization strategies a state chooses, it is 
unlikely that one reporting and dissemination tool, even 

a fully interactive, dynamic tool, can represent the entire 
CTE ecosystem or meet the needs of every audience. When 
publicly reporting CTE data, states should clearly indicate 
where the user can get more information and provide links 
to other data tools or more in-depth information about CTE 
programs that will add meaning for different audiences. 


Median Wages 


Source: RI DataHUB® 
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9. Help Users Interpret Data and Take Action 


KEY 
QUESTIONS 
TO CONSIDER 


Does your CTE data reporting include takeaways that 
clearly articulate what a user is seeing in a particular 
visualization? Have you consulted stakeholders 

to ensure these takeaways make sense to a lay 
audience? 


How well does your reporting explain to users the 


questions that can and cannot be answered by 
the data? 


Are there appropriate actions that you can ask users 
to take after using data reporting and dissemination 
tools? 


In addition to facilitating comparisons and customization, 

CTE leaders can help users interpret data and understand the 
value of CTE access and performance measures. Users have 
different levels of data fluency, and many of these design 
principles highlight the need to help all stakeholders — 
including learners and their families — better use CTE data 
reporting and understand its content. And while educators 
and some policymakers may see education statistics on a daily 
basis, even internal stakeholders will have different levels of 
expertise. 


Data interpretation can help stakeholders, particularly 

those new to CTE, understand the value of CTE and make 
sense of performance indicators, including those required 
by Perkins V. Features such as captions, popout boxes or 
tooltips can not only be used for indicator definitions or 
technical documentation, as described in Principle 4, but 
also help explain the value of monitoring CTE program 
concentration or describe what stakeholders can learn from 
the new Perkins V secondary program quality indicators. This 


interpretative assistance can be embedded within data tools 
and visualizations or offered through supplemental tools 
such as a brief webinar that provides additional framing and 
background information. 


Interpretation can also include summarizing data into more 
understandable takeaways. For instance, a chart showing CTE 
concentrator populations disaggregated by gender could 

be accompanied by a caption, popout box or tooltip that 
distills the information in the chart into a single statement, 
such as “Female learners in [state name] were more likely to 
be CTE concentrators in the Health Sciences Career Cluster 

in the 2018-19 school year than male learners.’ This element 
of the visualization can be static or, if using a customizable 
dashboard, based on selections made by the user. 


More interpretative elements of data tools can also tell users 
which questions can — and cannot — be answered by 

the data and even guide users to take action. For instance, 
reporting tools based on student performance data can 
direct users to other sources for data on program quality 

and implementation. Reporting and dissemination tools can 
also link users to next steps such as visiting a CTE program or 
taking part in CTE Month. 


While incorporating interpretation or action items into data 
reporting and dissemination can be powerful, CTE leaders 
should deploy these techniques cautiously. Interpretation 
should be specific and accurately reflect the data presented. 
Possible action items should be voluntary and non-partisan 
and should align with the state’s CTE goals. As with other 
design principles, stakeholder input can ensure that 
interpretations reflect a consensus on the meaning of the data 
and that action items are both meaningful and appropriate. 
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Conclusion 


Available, accessible and easy-to-use data reporting and 
dissemination tools are critical to meeting the needs of CTE 
learners, families, practitioners, policymakers and the public. 
The design principles described in this report can help state 
leaders develop data tools that support different audiences to 
access information, make decisions and take action. 
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